Title: The Effect of iMentor’s College Ready Program on High School Students’ College 
Aspirations and Non-Cognitive Skills 


Authors and Affiliations: Lisa Merrill, Nina Siman, David Kang, Jasmine Soltani, and Suzanne 
Wulaeh, 

Researeh Allianee for New York City Sehools, New York University 


SREE Spring 2015 Conference Abstract 



Background / Context: 

A growing body of research shows that school-based mentoring programs can be a flexible and 
cost-effective way to improve student outcomes (Angrist et ah, 2009). Effective mentoring 
programs create close bonds between students and caring adults, providing students with an 
important source of emotional support (Deutsch & Spencer, 2009; Spencer & Rhodes, 2005). 
Research shows that mentoring programs typically have greater effects on non-academic areas 
like self-esteem and attitudinal changes than on academic outcomes like test scores (Wood & 
Mayo-Wilson, 2012; Herrera et ah, 2007). However, mentors can provide important motivation 
for students by highlighting the importance of succeeding in school and showing how academic 
skills can be used in the real world (Bayer et ah, 2013). 

Of course, mentoring programs come in many shapes and sizes. Researchers have identified 
three important characteristics that can make some mentoring programs more successful than 
others. First, programs that carefully match mentees and mentors based on similar interests are 
more successful (Ensher & Murphy, 1997; Madia & Eutz, 2004). Second, mentors who are well 
trained offer better support (MENTOR, 2009). Finally, programs that monitor and nurture 
mentor-mentee relationships over multiple years through programmatic supports are particularly 
successful (DuBois et ah, 2002; Herrera et ah, 2000; Rhodes et ah, 2005). 

This study tests the effects of a new model — ^whole school-based mentoring that aims to prepare 
students to be college-ready. The whole-school model enables iMentor to enhance the effects of 
the mentoring relationships with a structured, college-readiness curricular component. In this 
study, we follow 16 cohorts of 9**^ grade students participating in iMentor over five years 
beginning in 2012-13 school year. This paper provides results from the 9* grade year. 

Purpose / Objective / Research Question / Focus of Study: 

The focus of this study is to identify the effects of the iMentor program after being exposed to 
one year of treatment. 

Research Questions: 

1. Do the iMentor evaluation schools implement iMentor’s College Ready Program with 
fidelity to the program model? 

2. What are the effects of iMentor on students’ non-academic outcomes, such as building and 
maintaining strong adult relationships, college aspirations, and non-cognitive skills? 

3. Do the overall effects of iMentor on students’ outcomes depend on the relative intensity and 
quality of various components of the iMentor program? 

Setting: 

iMentor is being implemented in eight public, non-charter high schools in New York City, the 
largest school district in the country. With over 1,600 schools, 1.1 million students, and 75,000 
teachers, the district is characterized by a wide range of school and student characteristics. 
iMentor recruited these eight schools because they serve low-income students. These schools 
have varying performance levels, student demographics, and have been in operation for a 
varying amount of time. (See table 1) 
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Population / Participants / Subjects: 

The population consists of 300 9* grade students in 3 schools over the 2012-13 school year, and 
800 9* grade students in 8 schools in the 2013-14 school year. The characteristics of the 
evaluation schools in comparison to the rest of NYC’s public high schools are in Table 1. 

Intervention / Program / Practice: 

The iMentor College Ready Program 

IMentor’s College Ready Program is a four-year intervention that seeks to promote college 
enrollment and persistence. The program attempts to recruit every member of a given 9* grade 
cohort and match each student with his or her own college-educated, adult mentor, with whom 
they are supposed to communicate online at least weekly and meet in person monthly. 

In addition to fostering a mentoring relationship, iMentor staff teach the iMentor college ready 
curriculum weekly and organizing monthly curricular events. During class, students email their 
mentors about the day’s lesson. Student meet with their mentors during iMentor events that are 
designed to align with the iMentor curriculum. 

iMentor supports the pairs and works to integrate its program into partner schools’ by providing 
a full-time iMentor employee for each cohort of students. This employee leads the weekly 
iMentor class (with a school staff member in the room to help facilitate), plans and manages 
events, and supports pairs. 

Research Design: 

Implementation 

To study the implementation fidelity we measure core program elements (student emailing, class 
attendance, event attendance, and mentor-mentee match consistency) and determine if the 
iMentor College Ready Program occurred in the school sites. We use continuous measures of 
each program component as well as relationship strength to study the variation in program 
intensity. We depend on program data as well as student surveys to create implementation 
measures. 

Non-Academic Outcomes 

Our analysis of non-academic outcomes relies on student survey data collected specifically for 
the iMentor evaluation. We will focus on nine measures, which are aligned with the core non- 
academic outcomes that iMentor aims to achieve for all students enrolled in the College Ready 
Program. The nine measures are (1) Growth Mindset, (2) Perseverance, (3) Optimism, (4) 

Critical Thinking, (5) Self Advocacy, (6) Help Seeking, and (7) Social Capital, (8) Strong 
Relationships, and (9) College Aspirations. 

This study uses a lagged cohort design with two treatment cohorts — essentially a double lagged 
design. To that end, all of the comparison students and students iMentor intends to treat attend 
these eight schools. The first cohort is a comparison cohort who entered one of the eight iMentor 
evaluation schools prior to implementing iMentor. The next cohort was treated with iMentor, and 
then there is a second lagged cohort in the same school designed to increase the power of the 
study. These lagged cohorts are staggered that phases in programmatic scale-up. Three schools 
began treating students with iMentor in 2012-13, and five schools began treating students in 
2013-14. In total, there are about 800 comparison students and 1,100 treatment students. The 
strength of this design rests on the assumption that cohorts are well-matched — that 9*’’ graders 
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entering a sehool in one year are similar to 9* graders the following year, or that differenees ean 
be eontrolled for using available data. One weakness of this design is that we cannot differentiate 
between the effects of iMentor and time or unobservable differences between cohorts. 

Data Collection and Analysis: 

Programmatic Data: 

iMentor collects data from mentees and mentors via iMi, their interactive online platform. This 
platform requires a username and password and is accessible to mentees and mentors, as well as 
iMentor staff and administration. For mentees and mentors, iMi is largely a place to send and 
receive emails, fill out surveys, and receive and respond to iMentor event invitations. iMentor 
staff use iMi to access data including information about student participation in iMentor class, 
logs of sent and received emails, and event attendance. 

Administrative Data: 

The Research Alliance for New York City Schools has created a rich administrative database 
about New York City students, schools, and educators. The individual-level data provide 
covariate controls to ensure statistical parity between the comparison and treatment cohorts. 
Variables include student demographic characteristics, 8* grade achievement scores, and S*** 
grade attendance. 

Survey Data 

Students in iMentor schools take a baseline survey in the fall of 9**^ grade (before mentors and 
mentees are matched), and complete another survey each spring. The student survey contains 
over 100 items, including measures of non-academic outcomes, their experience with mentors, as 
well as details about their background that cannot be obtained with administrative data (e.g., 
parent education level). 

Analysis 

Cronbach's alpha and factor analysis show that the theoretical constructions are valid and 
reliable. They have high internal correlation and are correlated with other desirable outcomes 
such as passing the end of year exam (New York State Algebra Regents). 

For each of the 9 outcomes, we will build a multi-level model with year fixed effects that 
accounts for non-independence of error of students in the same school. We will correct for 
multiple -hypothesis testing using the Benjamini-Hochberg correction as recommended by the 
What Works Clearinghouse’s Procedures and Standards Handbook, 3.0. 


Model: 

Social Emotional Effects: 

Yij = I NT + p^IMNj + P2STUCOVij + p^COHCOVj + p^SCHj + ej + Cij 
Yij = non-academic outcome for student I in cohort) 

IMNj= 1 if cohort is participating in iMentor, 0 if comparison cohort 
STUCOVij= series of student background covariates such as baseline non-academic 
characteristics, prior test scores, SES, parental background, and student demographics. 
Depending on the variables, these may be group mean centered if the cohort average is used at a 
covariate at the cohort level of analysis. 
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COHCOVj= series of eohort baekground covariates such as average baseline characteristics and 
average prior student test scores, and demographic characteristics for the cohort. 

SCHj= indicates that we will be using a school fixed-effects model — all estimates will be within 
school estimate 

6j = error associated with the cohort 
6ij = unexplained error 

Findings / Results: 

The analysis is currently underway and will be completed by January 2015, in time for a 
published report in March. 

Conclusions: 

Description of conclusions , recommendations, and limitations based on findings . 

It is important to note some limitations to our design. First, our study of non-academic outcomes 
does not allow us to differentiate between the effects of iMentor and time. For example, if later 
cohorts are doing better than the prior cohorts, we cannot be certain that these differences are due 
to iMentor exclusively. Second, we cannot control for other initiatives that aim to improve 
college readiness and are introduced at the same time as iMentor. Thus, we must be cautious 
about drawing causal inferences from these analyses. 

Despite these limitations, we are confident that our analyses will provide valuable insights and 
lessons — for mentoring programs, schools and districts — about the potential for using iMentor’s 
model to impact important nonacademic outcomes. 
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Appendix B. Tables and Figures 

Table 1 : Demographic Profile of iMentor College Ready Program Evaluation Schools and 

All Other NYC High Schools, 2011-2012 

Other NYC Evaluation 

High Schools^ Schools 


Gender (%) 


Female 

SI. 3 

54.0 

Male 

48.7 

46.0 

Race (%) 



Latino 

43.3 

SS.O 

Black 

38.4 

38.4 

White 

7.5 

2.4 

Aslan 

9.5 

2.7 

Receiving speciai education services (%) 

15.0 

12.8 

Engiish ianguage iearners (%) 

12.7 

19.7 

Poverty '’(%) 

72.0 

81.1 

Students per schooi 
Totai number of schoois 
Totai number of students 

553.7 

460 

254,706 

326.5' 

8 

2,612 


Source: Research Alliance calculations using data provided by the NYC DOE. 

Notes: ® High schools are defined as any school serving students grades 9-12; table does not include District 79, District 75, or 
specialized high schools. Total number of other NYC high schools varies between Tables 2, 4, and 5 due to data availability for 
different student characteristics..*’ Includes students who turned in their free or reduced price lunch form and those who did not turn 
in their form but attend a school that receives universal free lunch. Many students who are eligible for free or reduced lunch do not 
turn in their forms, therefore including universal programs is a more accurate measure of poverty. ‘’The slight discrepancy between 
the calculated number of students based on the listed average school size and the total number of schools due to rounding. 
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